{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1fe6e643-9453-4381-9445-bd471685fb96",
   "metadata": {},
   "source": [
    "# Labeling the [banking](https://huggingface.co/datasets/banking77) dataset using Autolabel\n",
    "\n",
    "This is a multi-class classification task where the input are customer service queries and we have to correctly label them with one of 77 intents. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "aacac4ae-c7f9-4dee-be3a-a2a6bfa099fa",
   "metadata": {},
   "source": [
    "## Install Autolabel\n",
    "Plus, setup your OpenAI API key, since we'll be using `gpt-3.5-turbo` as our LLM for labeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3dc19059-2f63-44b7-9a32-8b38a11249aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install 'refuel-autolabel[openai]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fbdeca2f-dd20-4634-b3f8-1b3ed45c4705",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# provide your own OpenAI API key here\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-XXXXXXXXXXXXXXXXXXXXXXXX\"\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8eb09918-494a-42ef-86a7-df93b3ce4284",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Download the dataset\n",
    "\n",
    "This dataset is available to install via Autolabel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b4c116ce-3294-45c2-9158-eac929f03a95",
   "metadata": {},
   "outputs": [],
   "source": [
    "from autolabel import get_data\n",
    "\n",
    "get_data(\"banking\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5f038fcc-9ec0-4f2d-b84d-2e963656c6bd",
   "metadata": {},
   "source": [
    "This downloads two datasets:\n",
    "* `test.csv`: This is the larger dataset we are trying to label using LLMs\n",
    "* `seed.csv`: This is a small dataset where we already have human-provided labels"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "84b014d1-f45c-4479-9acc-0d20870b1786",
   "metadata": {},
   "source": [
    "## Start the labeling process!\n",
    "\n",
    "Labeling with Autolabel is a 3-step process:\n",
    "* First, we specify a labeling configuration (see `config.json` below)\n",
    "* Next, we do a dry-run on our dataset using the LLM specified in `config.json` by running `agent.plan`\n",
    "* Finally, we run the labeling with `agent.run`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "33d47b67-0718-4289-bb59-989d851d09ed",
   "metadata": {},
   "source": [
    "### First labeling run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c093fe91-3508-4140-8bd6-217034e3cce6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "from autolabel import LabelingAgent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c93fae0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the config\n",
    "with open(\"config_banking.json\") as f:\n",
    "     config = json.load(f)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1fad4e85-d598-413d-9b01-9690653a05ad",
   "metadata": {},
   "source": [
    "Let's review the configuration file below. You'll notice the following useful keys:\n",
    "* `task_type`: `classification` (since it's a classification task)\n",
    "* `model`: `{'provider': 'openai', 'name': 'gpt-3.5-turbo'}` (use a specific OpenAI model)\n",
    "* `prompt.task_guidelines`: `'You are an expert at understanding bank customers support complaints and queries...` (how we describe the task to the LLM)\n",
    "* `prompt.labels`: `['age_limit', 'apple_pay_or_google_pay', 'atm_support', ...]` (the full list of labels to choose from)\n",
    "* `prompt.few_shot_num`: 10 (how many labeled examples to provide to the LLM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6a3610fd-721e-44de-9b2c-2cd73ec86bba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'task_name': 'BankingComplaintsClassification',\n",
       " 'task_type': 'classification',\n",
       " 'dataset': {'label_column': 'label', 'delimiter': ','},\n",
       " 'model': {'provider': 'openai', 'name': 'gpt-3.5-turbo'},\n",
       " 'prompt': {'task_guidelines': 'You are an expert at understanding bank customers support complaints and queries.\\nYour job is to correctly classify the provided input example into one of the following categories.\\nCategories:\\n{labels}',\n",
       "  'output_guidelines': 'You will answer with just the the correct output label and nothing else.',\n",
       "  'labels': ['activate_my_card',\n",
       "   'age_limit',\n",
       "   'apple_pay_or_google_pay',\n",
       "   'atm_support',\n",
       "   'automatic_top_up',\n",
       "   'balance_not_updated_after_bank_transfer',\n",
       "   'balance_not_updated_after_cheque_or_cash_deposit',\n",
       "   'beneficiary_not_allowed',\n",
       "   'cancel_transfer',\n",
       "   'card_about_to_expire',\n",
       "   'card_acceptance',\n",
       "   'card_arrival',\n",
       "   'card_delivery_estimate',\n",
       "   'card_linking',\n",
       "   'card_not_working',\n",
       "   'card_payment_fee_charged',\n",
       "   'card_payment_not_recognised',\n",
       "   'card_payment_wrong_exchange_rate',\n",
       "   'card_swallowed',\n",
       "   'cash_withdrawal_charge',\n",
       "   'cash_withdrawal_not_recognised',\n",
       "   'change_pin',\n",
       "   'compromised_card',\n",
       "   'contactless_not_working',\n",
       "   'country_support',\n",
       "   'declined_card_payment',\n",
       "   'declined_cash_withdrawal',\n",
       "   'declined_transfer',\n",
       "   'direct_debit_payment_not_recognised',\n",
       "   'disposable_card_limits',\n",
       "   'edit_personal_details',\n",
       "   'exchange_charge',\n",
       "   'exchange_rate',\n",
       "   'exchange_via_app',\n",
       "   'extra_charge_on_statement',\n",
       "   'failed_transfer',\n",
       "   'fiat_currency_support',\n",
       "   'get_disposable_virtual_card',\n",
       "   'get_physical_card',\n",
       "   'getting_spare_card',\n",
       "   'getting_virtual_card',\n",
       "   'lost_or_stolen_card',\n",
       "   'lost_or_stolen_phone',\n",
       "   'order_physical_card',\n",
       "   'passcode_forgotten',\n",
       "   'pending_card_payment',\n",
       "   'pending_cash_withdrawal',\n",
       "   'pending_top_up',\n",
       "   'pending_transfer',\n",
       "   'pin_blocked',\n",
       "   'receiving_money',\n",
       "   'Refund_not_showing_up',\n",
       "   'request_refund',\n",
       "   'reverted_card_payment?',\n",
       "   'supported_cards_and_currencies',\n",
       "   'terminate_account',\n",
       "   'top_up_by_bank_transfer_charge',\n",
       "   'top_up_by_card_charge',\n",
       "   'top_up_by_cash_or_cheque',\n",
       "   'top_up_failed',\n",
       "   'top_up_limits',\n",
       "   'top_up_reverted',\n",
       "   'topping_up_by_card',\n",
       "   'transaction_charged_twice',\n",
       "   'transfer_fee_charged',\n",
       "   'transfer_into_account',\n",
       "   'transfer_not_received_by_recipient',\n",
       "   'transfer_timing',\n",
       "   'unable_to_verify_identity',\n",
       "   'verify_my_identity',\n",
       "   'verify_source_of_funds',\n",
       "   'verify_top_up',\n",
       "   'virtual_card_not_working',\n",
       "   'visa_or_mastercard',\n",
       "   'why_verify_identity',\n",
       "   'wrong_amount_of_cash_received',\n",
       "   'wrong_exchange_rate_for_cash_withdrawal'],\n",
       "  'few_shot_examples': 'seed.csv',\n",
       "  'few_shot_selection': 'semantic_similarity',\n",
       "  'few_shot_num': 10,\n",
       "  'example_template': 'Input: {example}\\nOutput: {label}'}}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "acb4a3de-fa84-4b94-b17a-7a6fac892a1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an agent for labeling\n",
    "agent = LabelingAgent(config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "92667a39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "22356178f4bd4e7cab7e390da7ed0543",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┌──────────────────────────┬─────────┐\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Total Estimated Cost     </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $6.0123 </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Number of Examples       </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> 1998    </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Average cost per example </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $0.003  </span>│\n",
       "└──────────────────────────┴─────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┌──────────────────────────┬─────────┐\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mTotal Estimated Cost    \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$6.0123\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mNumber of Examples      \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m1998   \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mAverage cost per example\u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$0.003 \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "└──────────────────────────┴─────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────── </span>Prompt Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────── \u001b[0mPrompt Example\u001b[92m ──────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are an expert at understanding bank customers support complaints and queries.\n",
      "Your job is to correctly classify the provided input example into one of the following categories.\n",
      "Categories:\n",
      "activate_my_card\n",
      "age_limit\n",
      "apple_pay_or_google_pay\n",
      "atm_support\n",
      "automatic_top_up\n",
      "balance_not_updated_after_bank_transfer\n",
      "balance_not_updated_after_cheque_or_cash_deposit\n",
      "beneficiary_not_allowed\n",
      "cancel_transfer\n",
      "card_about_to_expire\n",
      "card_acceptance\n",
      "card_arrival\n",
      "card_delivery_estimate\n",
      "card_linking\n",
      "card_not_working\n",
      "card_payment_fee_charged\n",
      "card_payment_not_recognised\n",
      "card_payment_wrong_exchange_rate\n",
      "card_swallowed\n",
      "cash_withdrawal_charge\n",
      "cash_withdrawal_not_recognised\n",
      "change_pin\n",
      "compromised_card\n",
      "contactless_not_working\n",
      "country_support\n",
      "declined_card_payment\n",
      "declined_cash_withdrawal\n",
      "declined_transfer\n",
      "direct_debit_payment_not_recognised\n",
      "disposable_card_limits\n",
      "edit_personal_details\n",
      "exchange_charge\n",
      "exchange_rate\n",
      "exchange_via_app\n",
      "extra_charge_on_statement\n",
      "failed_transfer\n",
      "fiat_currency_support\n",
      "get_disposable_virtual_card\n",
      "get_physical_card\n",
      "getting_spare_card\n",
      "getting_virtual_card\n",
      "lost_or_stolen_card\n",
      "lost_or_stolen_phone\n",
      "order_physical_card\n",
      "passcode_forgotten\n",
      "pending_card_payment\n",
      "pending_cash_withdrawal\n",
      "pending_top_up\n",
      "pending_transfer\n",
      "pin_blocked\n",
      "receiving_money\n",
      "Refund_not_showing_up\n",
      "request_refund\n",
      "reverted_card_payment?\n",
      "supported_cards_and_currencies\n",
      "terminate_account\n",
      "top_up_by_bank_transfer_charge\n",
      "top_up_by_card_charge\n",
      "top_up_by_cash_or_cheque\n",
      "top_up_failed\n",
      "top_up_limits\n",
      "top_up_reverted\n",
      "topping_up_by_card\n",
      "transaction_charged_twice\n",
      "transfer_fee_charged\n",
      "transfer_into_account\n",
      "transfer_not_received_by_recipient\n",
      "transfer_timing\n",
      "unable_to_verify_identity\n",
      "verify_my_identity\n",
      "verify_source_of_funds\n",
      "verify_top_up\n",
      "virtual_card_not_working\n",
      "visa_or_mastercard\n",
      "why_verify_identity\n",
      "wrong_amount_of_cash_received\n",
      "wrong_exchange_rate_for_cash_withdrawal\n",
      "\n",
      "You will answer with just the the correct output label and nothing else.\n",
      "\n",
      "Some examples with their output answers are provided below:\n",
      "\n",
      "Input: I would like to delete my account please.\n",
      "Output: terminate_account\n",
      "\n",
      "Input: Can you help me get rid of my account?\n",
      "Output: terminate_account\n",
      "\n",
      "Input: Help me cancel my transaction\n",
      "Output: cancel_transfer\n",
      "\n",
      "Input: I need to cancel a purchase I made.\n",
      "Output: request_refund\n",
      "\n",
      "Input: I want to change my personal details.\n",
      "Output: edit_personal_details\n",
      "\n",
      "Input: How can I withdraw money?\n",
      "Output: atm_support\n",
      "\n",
      "Input: How do I do a successful transfer to an account?\n",
      "Output: beneficiary_not_allowed\n",
      "\n",
      "Input: Why wasn't I able to transfer to another account?\n",
      "Output: beneficiary_not_allowed\n",
      "\n",
      "Input: Can my daughter open an account?\n",
      "Output: age_limit\n",
      "\n",
      "Input: I need to update my current address\n",
      "Output: edit_personal_details\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: I want to close my account\n",
      "Output: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# dry-run -- this tells us how much this will cost and shows an example prompt\n",
    "from autolabel import AutolabelDataset\n",
    "\n",
    "ds = AutolabelDataset(\"data/banking/test.csv\", config=config)\n",
    "agent.plan(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "dd703025-54d8-4349-b0d6-736d2380e966",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8e824273466344469a571964c95e5507",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:26:53 autolabel.tasks.base WARNING: LLM response is not in the labels list\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:26:57 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88913 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:26:57 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88913 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:26:58 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88462 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:26:58 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88462 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:00 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89189 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:00 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89189 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:01 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88721 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:01 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88721 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:03 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89387 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:03 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89387 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:05 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88905 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:05 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88905 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:06 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88502 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:06 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88502 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:08 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89068 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:08 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89068 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:10 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88677 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:10 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88677 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:11 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88274 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:11 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88274 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:13 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88551 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:13 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88551 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:15 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89386 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:15 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89386 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:19 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89026 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:19 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89026 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual Cost: 0.1017\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.7677   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.99            </span>│\n",
       "└──────────┴─────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.7677  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.99           \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└──────────┴─────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# now, do the actual labeling\n",
    "ds = agent.run(ds, max_items=100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "73037ec4-0282-4058-8336-c81f6dc6e711",
   "metadata": {},
   "source": [
    "We are at 76% accuracy when labeling the first 100 examples. Let's see if we can use confidence scores to improve accuracy further by removing the less confident examples from our labeled set."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4d7645ab",
   "metadata": {},
   "source": [
    "## Compute confidence scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "5fbc1264",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start computing confidence scores (using Refuel's LLMs)\n",
    "os.environ[\"REFUEL_API_KEY\"] = \"sk-xxxxxxxxxxxx\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "8a923034-b41b-47bb-91a7-7332a33f151f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# set `compute_confidence` -> True\n",
    "config[\"model\"][\"compute_confidence\"] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "1998f5e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = LabelingAgent(config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "119e6f22",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating Prompts... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100/100 0:00:24 0:00:00\n",
      "┌──────────────────────────┬─────────┐\n",
      "│ Total Estimated Cost     │ $6.6836 │\n",
      "│ Number of Examples       │ 1998    │\n",
      "│ Average cost per example │ $0.0033 │\n",
      "└──────────────────────────┴─────────┘\n",
      "──────────────────────────────── Prompt Example ────────────────────────────────\n",
      "You are an expert at understanding bank customers support complaints and queries.\n",
      "Your job is to correctly classify the provided input example into one of the following categories.\n",
      "Categories:\n",
      "activate_my_card\n",
      "age_limit\n",
      "apple_pay_or_google_pay\n",
      "atm_support\n",
      "automatic_top_up\n",
      "balance_not_updated_after_bank_transfer\n",
      "balance_not_updated_after_cheque_or_cash_deposit\n",
      "beneficiary_not_allowed\n",
      "cancel_transfer\n",
      "card_about_to_expire\n",
      "card_acceptance\n",
      "card_arrival\n",
      "card_delivery_estimate\n",
      "card_linking\n",
      "card_not_working\n",
      "card_payment_fee_charged\n",
      "card_payment_not_recognised\n",
      "card_payment_wrong_exchange_rate\n",
      "card_swallowed\n",
      "cash_withdrawal_charge\n",
      "cash_withdrawal_not_recognised\n",
      "change_pin\n",
      "compromised_card\n",
      "contactless_not_working\n",
      "country_support\n",
      "declined_card_payment\n",
      "declined_cash_withdrawal\n",
      "declined_transfer\n",
      "direct_debit_payment_not_recognised\n",
      "disposable_card_limits\n",
      "edit_personal_details\n",
      "exchange_charge\n",
      "exchange_rate\n",
      "exchange_via_app\n",
      "extra_charge_on_statement\n",
      "failed_transfer\n",
      "fiat_currency_support\n",
      "get_disposable_virtual_card\n",
      "get_physical_card\n",
      "getting_spare_card\n",
      "getting_virtual_card\n",
      "lost_or_stolen_card\n",
      "lost_or_stolen_phone\n",
      "order_physical_card\n",
      "passcode_forgotten\n",
      "pending_card_payment\n",
      "pending_cash_withdrawal\n",
      "pending_top_up\n",
      "pending_transfer\n",
      "pin_blocked\n",
      "receiving_money\n",
      "Refund_not_showing_up\n",
      "request_refund\n",
      "reverted_card_payment?\n",
      "supported_cards_and_currencies\n",
      "terminate_account\n",
      "top_up_by_bank_transfer_charge\n",
      "top_up_by_card_charge\n",
      "top_up_by_cash_or_cheque\n",
      "top_up_failed\n",
      "top_up_limits\n",
      "top_up_reverted\n",
      "topping_up_by_card\n",
      "transaction_charged_twice\n",
      "transfer_fee_charged\n",
      "transfer_into_account\n",
      "transfer_not_received_by_recipient\n",
      "transfer_timing\n",
      "unable_to_verify_identity\n",
      "verify_my_identity\n",
      "verify_source_of_funds\n",
      "verify_top_up\n",
      "virtual_card_not_working\n",
      "visa_or_mastercard\n",
      "why_verify_identity\n",
      "wrong_amount_of_cash_received\n",
      "wrong_exchange_rate_for_cash_withdrawal\n",
      "\n",
      "You will answer with just the the correct output label and nothing else.\n",
      "\n",
      "Some examples with their output answers are provided below:\n",
      "\n",
      "Input: I would like to delete my account please.\n",
      "Output: terminate_account\n",
      "\n",
      "Input: Can you help me get rid of my account?\n",
      "Output: terminate_account\n",
      "\n",
      "Input: Help me cancel my transaction\n",
      "Output: cancel_transfer\n",
      "\n",
      "Input: I need to cancel a purchase I made.\n",
      "Output: request_refund\n",
      "\n",
      "Input: I want to change my personal details.\n",
      "Output: edit_personal_details\n",
      "\n",
      "Input: How can I withdraw money?\n",
      "Output: atm_support\n",
      "\n",
      "Input: How do I do a successful transfer to an account?\n",
      "Output: beneficiary_not_allowed\n",
      "\n",
      "Input: Why wasn't I able to transfer to another account?\n",
      "Output: beneficiary_not_allowed\n",
      "\n",
      "Input: Can my daughter open an account?\n",
      "Output: age_limit\n",
      "\n",
      "Input: I need to update my current address\n",
      "Output: edit_personal_details\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: I want to close my account\n",
      "Output: \n",
      "────────────────────────────────────────────────────────────────────────────────\n"
     ]
    }
   ],
   "source": [
    "from autolabel import AutolabelDataset\n",
    "\n",
    "ds = AutolabelDataset(\"data/banking/test.csv\", config=config)\n",
    "agent.plan(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "63c74705",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "84ad306191e44a60918b09abae371b68",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-13 22:59:21 openai INFO: error_code=None error_message='That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID 603d90627ab4c936108e1009bec434b8 in your message.)' error_param=None error_type=server_error message='OpenAI API error received' stream_error=False\n",
      "2023-06-13 22:59:21 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID 603d90627ab4c936108e1009bec434b8 in your message.).\n",
      "2023-06-13 23:00:06 openai INFO: error_code=None error_message='That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID 9ddebe0bfe2beb3935b21d667d5905ec in your message.)' error_param=None error_type=server_error message='OpenAI API error received' stream_error=False\n",
      "2023-06-13 23:00:06 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID 9ddebe0bfe2beb3935b21d667d5905ec in your message.).\n",
      "2023-06-13 23:01:04 openai INFO: error_code=None error_message='That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID 109247fedbf578ce085155e22c6f5196 in your message.)' error_param=None error_type=server_error message='OpenAI API error received' stream_error=False\n",
      "2023-06-13 23:01:04 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID 109247fedbf578ce085155e22c6f5196 in your message.).\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metric: auroc: 0.8737\n",
      "Actual Cost: 0.1356\n",
      "┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
      "┃ support ┃ threshold ┃ accuracy ┃ completion_rate ┃\n",
      "┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
      "│ 100     │ -inf      │ 0.74     │ 1.0             │\n",
      "│ 1       │ 0.9999    │ 1.0      │ 0.01            │\n",
      "│ 12      │ 0.9992    │ 1.0      │ 0.12            │\n",
      "│ 13      │ 0.9991    │ 0.9231   │ 0.13            │\n",
      "│ 41      │ 0.9916    │ 0.9756   │ 0.41            │\n",
      "│ 42      │ 0.9912    │ 0.9524   │ 0.42            │\n",
      "│ 43      │ 0.9912    │ 0.9535   │ 0.43            │\n",
      "│ 44      │ 0.9901    │ 0.9318   │ 0.44            │\n",
      "│ 48      │ 0.9873    │ 0.9375   │ 0.48            │\n",
      "│ 49      │ 0.9872    │ 0.9184   │ 0.49            │\n",
      "│ 63      │ 0.9695    │ 0.9365   │ 0.63            │\n",
      "│ 64      │ 0.9664    │ 0.9219   │ 0.64            │\n",
      "│ 66      │ 0.9601    │ 0.9242   │ 0.66            │\n",
      "│ 67      │ 0.9587    │ 0.9104   │ 0.67            │\n",
      "│ 68      │ 0.9523    │ 0.9118   │ 0.68            │\n",
      "│ 69      │ 0.95      │ 0.8986   │ 0.69            │\n",
      "│ 74      │ 0.9305    │ 0.9054   │ 0.74            │\n",
      "│ 75      │ 0.9066    │ 0.8933   │ 0.75            │\n",
      "│ 76      │ 0.905     │ 0.8947   │ 0.76            │\n",
      "│ 77      │ 0.9046    │ 0.8831   │ 0.77            │\n",
      "│ 78      │ 0.8996    │ 0.8846   │ 0.78            │\n",
      "│ 81      │ 0.8786    │ 0.8519   │ 0.81            │\n",
      "│ 82      │ 0.8764    │ 0.8537   │ 0.82            │\n",
      "│ 84      │ 0.8657    │ 0.8333   │ 0.84            │\n",
      "│ 86      │ 0.8547    │ 0.8372   │ 0.86            │\n",
      "│ 89      │ 0.8004    │ 0.809    │ 0.89            │\n",
      "│ 90      │ 0.7594    │ 0.8111   │ 0.9             │\n",
      "│ 93      │ 0.7036    │ 0.7849   │ 0.93            │\n",
      "│ 94      │ 0.6932    │ 0.7872   │ 0.94            │\n",
      "│ 100     │ 0.2895    │ 0.74     │ 1.0             │\n",
      "└─────────┴───────────┴──────────┴─────────────────┘\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Total number of failures: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Total number of failures: \u001b[1;36m0\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds = agent.run(ds, max_items=100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "62dc2f6f-29e0-4b32-be7b-e10d99698c51",
   "metadata": {},
   "source": [
    "Looking at the table above, we can see that if we set the confidence threshold at `0.9305`, we are able to label at 90% accuracy and getting a completion rate of 74%. This means, we would ignore all the data points where confidence score is less than `0.9305` (which would end up being around 26% of all samples). This would, however, guarantee a very high quality labeled dataset for us. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
